Skip to content

UPSTREAM: <carry>: fix kubernetes/conformance to filter on [Conformance] label#2694

Open
not-stbenjam wants to merge 1 commit into
openshift:masterfrom
not-stbenjam:fix-kubernetes-conformance-suite
Open

UPSTREAM: <carry>: fix kubernetes/conformance to filter on [Conformance] label#2694
not-stbenjam wants to merge 1 commit into
openshift:masterfrom
not-stbenjam:fix-kubernetes-conformance-suite

Conversation

@not-stbenjam

@not-stbenjam not-stbenjam commented Jun 15, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown

Summary

  • Change kubernetes/conformance from AddGlobalSuite (parent umbrella, no qualifiers) to AddSuite with a [Conformance] label filter, so it only matches the 428 true upstream Kubernetes conformance tests instead of OCP's broader conformance view
  • Remove kubernetes/conformance from the Parents of kubernetes/conformance/parallel and kubernetes/conformance/serial so all suites are standalone
  • This suite is used by OCPT to run the minimal true upstream Kubernetes conformance tests

Test plan

  • Built k8s-tests-ext binary and verified kubernetes/conformance matches 428 tests (matching parallel/minimal 401 + serial/minimal 27)
  • Verify OCPT can run kubernetes/conformance suite successfully

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Refactor
    • Restructured conformance test suite registration to use explicit label qualifiers, improving how test qualifiers are inherited in serial conformance tests.

@not-stbenjam @claude
UPSTREAM: <carry>: fix kubernetes/conformance to filter on [Conforman…
7ffb8b2 
…ce] label

The kubernetes/conformance suite was incorrectly defined as an
AddGlobalSuite parent umbrella with no qualifiers, inheriting
everything from its children. This caused it to include OCP's broader
view of conformance rather than just the upstream Kubernetes conformance
tests.

Change it to a standalone AddSuite that filters on the Conformance
label, matching only the 428 true upstream conformance tests. Also
remove kubernetes/conformance from the Parents of the parallel and
serial suites so all three are standalone.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci-robot openshift-ci-robot added the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Jun 15, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 15, 2026

Copy link
Copy Markdown

@not-stbenjam: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

@coderabbitai

coderabbitai Bot commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d92d0723-28da-45f6-9d78-74cd82934d31

📥 Commits

Reviewing files that changed from the base of the PR and between d8d517e and 7ffb8b2.

📒 Files selected for processing (1)
  • openshift-hack/cmd/k8s-tests-ext/k8s-tests.go

Walkthrough

In openshift-hack/cmd/k8s-tests-ext/k8s-tests.go, the kubernetes/conformance suite is changed from an AddGlobalSuite umbrella registration with no qualifiers to a regular suite registration with an explicit Conformance label qualifier. The kubernetes/conformance/serial/minimal suite also has kubernetes/conformance removed from its parent suite list.

Changes

Conformance Suite Registration

Layer / File(s) Summary
kubernetes/conformance suite and qualifier wiring
openshift-hack/cmd/k8s-tests-ext/k8s-tests.go		 kubernetes/conformance is registered as a direct suite with a Conformance label qualifier instead of via AddGlobalSuite with no qualifiers; kubernetes/conformance is removed from the parent suite list of kubernetes/conformance/serial/minimal, changing how qualifier inheritance is applied for serial conformance tests.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: converting kubernetes/conformance to filter on [Conformance] label instead of using AddGlobalSuite.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All suite names in the PR are static and deterministic: "kubernetes/conformance", "kubernetes/conformance/parallel/minimal", etc. No dynamic values, timestamps, or generated identifiers present in...
Test Structure And Quality ✅ Passed The PR modifies a test suite configuration file (k8s-tests.go), not actual Ginkgo test code. The custom check is for reviewing Ginkgo test quality (single responsibility, setup/cleanup, timeouts, a...
Microshift Test Compatibility ✅ Passed PR does not add new Ginkgo e2e tests (It/Describe/Context/When). It only modifies test suite registration and filtering configuration in k8s-tests.go, making the MicroShift compatibility check not...
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR modifies only test suite configuration (filtering and registration) in k8s-tests.go, not Ginkgo test implementations. No new It(), Describe(), Context(), When() tests are added, so SNO comp...
Topology-Aware Scheduling Compatibility ✅ Passed PR modifies test suite registration (k8s-tests.go), not deployment manifests, operator code, or scheduling constraints. No pod scheduling, affinity, topology, or replica configuration changes present.
Ote Binary Stdout Contract ✅ Passed PR makes only structural Suite definition changes with no new stdout writes, logging calls, or Ginkgo configuration at process level. All changes are pure data assignments without side effects.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR modifies test suite definitions and label filters only, not adding new Ginkgo e2e tests. The check applies only when new test implementations are added.
No-Weak-Crypto ✅ Passed No weak cryptography patterns detected. The modified file is test configuration code with no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or insecure comparisons.
Container-Privileges ✅ Passed PR only modifies a Go test configuration file (k8s-tests.go), not Kubernetes manifests. The container-privileges check is not applicable to non-manifest code files.
No-Sensitive-Data-In-Logs ✅ Passed PR contains no changes to logging, error handling, or output code that would expose sensitive data (passwords, tokens, API keys, PII, etc.) in logs.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 15, 2026
@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown

Hi @not-stbenjam. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot requested review from jacobsee and jerpeter1 June 15, 2026 17:20
@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: not-stbenjam
Once this PR has been reviewed and has the lgtm label, please assign jerpeter1 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@stbenjam

stbenjam commented Jun 15, 2026

Copy link
Copy Markdown
Member

/ok-to-test

@openshift-ci openshift-ci Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 15, 2026
@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown

@not-stbenjam: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-crun 7ffb8b2 link true /test e2e-aws-ovn-crun
ci/prow/e2e-metal-ipi-ovn-ipv6 7ffb8b2 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-aws-ovn-techpreview-serial-1of2 7ffb8b2 link false /test e2e-aws-ovn-techpreview-serial-1of2
ci/prow/e2e-aws-ovn-runc 7ffb8b2 link false /test e2e-aws-ovn-runc
ci/prow/e2e-aws-ovn-serial-1of2 7ffb8b2 link true /test e2e-aws-ovn-serial-1of2
ci/prow/e2e-gcp 7ffb8b2 link true /test e2e-gcp
ci/prow/e2e-aws-ovn-fips 7ffb8b2 link true /test e2e-aws-ovn-fips
ci/prow/e2e-aws-ovn-techpreview 7ffb8b2 link false /test e2e-aws-ovn-techpreview
ci/prow/e2e-aws-ovn-cgroupsv2 7ffb8b2 link true /test e2e-aws-ovn-cgroupsv2

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@bshaw7

bshaw7 commented Jun 16, 2026

Copy link
Copy Markdown

/payload-job periodic-ci-openshift-release-main-nightly-5.0-opct-platform-external-aws-ccm periodic-ci-openshift-release-main-nightly-5.0-opct-platform-external-aws

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown

@bshaw7: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-opct-platform-external-aws-ccm
  • periodic-ci-openshift-release-main-nightly-5.0-opct-platform-external-aws

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/60bfb240-6972-11f1-8b90-c7abca0044c9-0

@bshaw7

bshaw7 commented Jun 16, 2026

Copy link
Copy Markdown

CI test shows, 10-openshift-kube-conformance count 436

Tue, 16 Jun 2026 17:03:39 UTC|4h31m0.043466768s> Global Status: complete
JOB_NAME                           | STATUS     | RESULTS    | PROGRESS                  | MESSAGE                                           
05-openshift-cluster-upgrade       | complete   | passed     | 1/1 (0 failures)          |                                                   
10-openshift-kube-conformance      | complete   | passed     | 401/436 (0 failures)      | Total tests processed: 406 (406 pass / 0 failed)  
20-openshift-conformance-validated | complete   | failed     | 4701/4701 (22 failures)   | Total tests processed: 2176 (2152 pass / 24 failed)
80-openshift-tests-replay          | complete   | failed     | 17/18 (16 failures)       | Total tests processed: 22 (5 pass / 17 failed)    
99-openshift-artifacts-collector   | complete   | passed     | 0/0 (0 failures)          | Total tests processed: 16 (16 pass / 0 failed)    
time="2026-06-16T17:03:39Z" level=info msg="The execution has completed! Use retrieve command to collect the results and share the archive with your Red Hat partner."
2026-06-16 11:36:19+00:00> Done with code=0

@bshaw7

bshaw7 commented Jun 18, 2026

Copy link
Copy Markdown

Local test also done,  here also it's listing 428 same as CI test.

Thu, 18 Jun 2026 08:41:12 UTC|32m32.395507637s> Global Status: running
JOB_NAME                           | STATUS     | RESULTS    | PROGRESS                  | MESSAGE
05-openshift-cluster-upgrade       | complete   |            | 1/1 (0 failures)          | complete
10-openshift-kube-conformance      | complete   |            | 428/428 (0 failures)      | complete
20-openshift-conformance-validated | running    |            | 5/4110 (0 failures)       | status=running=T/C/P/F/S=4110/5/2/0/3
80-openshift-tests-replay          | running    |            | 0/0 (0 failures)          | status=blocked-by=openshift-conformance-validated=(0/-4110/0)=[0/2000]
99-openshift-artifacts-collector   | running    |            | 0/0 (0 failures)          | status=blocked-by=openshift-tests-replay=(0/0/0)=[0/2000]
Thu, 18 Jun 2026 08:41:22 UTC|32m42.381460825s> Global Status: running
JOB_NAME                           | STATUS     | RESULTS    | PROGRESS                  | MESSAGE
05-openshift-cluster-upgrade       | complete   |            | 1/1 (0 failures)          | complete
10-openshift-kube-conformance      | complete   |            | 428/428 (0 failures)      | complete
20-openshift-conformance-validated | running    |            | 30/4110 (0 failures)      | status=running=T/C/P/F/S=4110/30/20/0/10
80-openshift-tests-replay          | running    |            | 0/0 (0 failures)          | status=waiting-for=openshift-conformance-validated=(0/-4080/0)=[0/2000]
99-openshift-artifacts-collector   | running    |            | 0/0 (0 failures)          | status=blocked-by=openshift-tests-replay=(0/0/0)=[0/2000]

[root@localhost ~]# oc get clusterversion
NAME      VERSION                                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   5.0.0-0-2026-06-17-171654-test-ci-ln-mgf24rt-latest   True        False         81m     Cluster version is 5.0.0-0-2026-06-17-171654-test-ci-ln-mgf24rt-latest
[root@localhost ~]#

Used release image which include 2694 PR fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jacobsee jacobsee Awaiting requested review from jacobsee

@jerpeter1 jerpeter1 Awaiting requested review from jerpeter1

Assignees

No one assigned

Labels

backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. ok-to-test Indicates a non-member PR verified by an org member that is safe to test.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@not-stbenjam @openshift-ci-robot @stbenjam @bshaw7